pkg/bpmn/jobs_api.go (1)

89-97: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

recordJobLifetime(..., "failed") also fires when the job was caught by an error boundary/subprocess, not genuinely failed.

This defer only checks retErr == nil, but JobFailByKey returns nil early both when the job is redirected to an error-boundary event (line ~111) or an error event-subprocess (line ~117), and when it's actually failed via failJobWithIncident (line ~127). Since the same defer fires on every nil-returning path, engine.metrics.JobsFailed and the new engine.recordJobLifetime(ctx, job, "failed") both misclassify caught/handled jobs as "failed", skewing the outcome-labeled telemetry this PR introduces.

Consider tracking whether the job was actually handled by a catch target and skipping the "failed" metrics in that case, or moving the metric calls to sit alongside the genuine failJobWithIncident success path instead of the blanket defer.

🐛 Sketch of a fix

+	handled := false
 	defer func() {
-		if retErr == nil {
+		if retErr == nil && !handled {
 			engine.metrics.JobsFailed.Add(ctx, 1, metric.WithAttributes(
 				attribute.String("type", job.Type),
 				attribute.Bool("internal", false),
 			))
 			engine.recordJobLifetime(ctx, job, "failed")
 		}
 	}()

 	if errorCode != nil {
 		target, err := engine.findErrorCatchTarget(ctx, &batch, instance, job.Token, errorCode)
 		if err != nil {
 			return err
 		}

 		if target != nil {
 			switch {
 			case target.boundary != nil:
 				if handledBoundary, err := engine.processBoundaryErrorEvent(ctx, &batch, job, instance, target.boundary, variables); err != nil {
 					return err
 				} else if handledBoundary {
+					handled = true
 					return nil
 				}
 			case target.eventSubprocess != nil:
 				if handledSub, err := engine.processErrorEventSubprocessForJob(ctx, &batch, job, target.eventSubprocess, variables); err != nil {
 					return err
 				} else if handledSub {
+					handled = true
 					return nil
 				}
 			}
 		}
 	}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/bpmn/jobs_api.go` around lines 89 - 97, Update JobFailByKey so caught
jobs are not classified as failed: distinguish error-boundary and
error-event-subprocess handling from the genuine failJobWithIncident path, and
emit engine.metrics.JobsFailed and engine.recordJobLifetime(ctx, job, "failed")
only for actual failures. Remove or adjust the blanket retErr-based defer while
preserving existing handling behavior.

scripts/grafana_provisioning/dashboards/zenbpm/cluster.json (2)

50-51: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not sum the per-node leader-change counter.

partition_leader_changes is recorded as an event count observed by each node. sum by(partition) counts the same event once per node. Select one authoritative series, or use max by(partition) when every node observes the same events.

Proposed query change

- "expr": "sum by(partition) (increase(partition_leader_changes_total[5m]))"
+ "expr": "max by(partition) (increase(partition_leader_changes_total[5m]))"

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/grafana_provisioning/dashboards/zenbpm/cluster.json` around lines 50
- 51, Update the dashboard target query for partition leader changes to avoid
summing duplicated per-node observations; replace sum by(partition) with an
authoritative single series or max by(partition), while preserving the existing
increase window, partition grouping, legendFormat, and refId.

---

21-22: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Use current Prometheus samples for status panels.

If metric collection stops, lastNotNull can display an old LEADER OK value across the one-hour range. Set "instant": true on both stat targets and configure a distinct no-data display.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/grafana_provisioning/dashboards/zenbpm/cluster.json` around lines 21
- 22, Update both status panel targets in the dashboard JSON to use current
Prometheus samples by setting instant querying on each target, and configure a
distinct no-data display for the stat panels. Preserve the existing expressions,
legends, and references while applying the same behavior consistently to both
targets.

internal/cluster/partition/partition.go (1)

87-102: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Log the metric that actually failed.

The processInstancesActive registration failure is reported as "Failed to register meter for jobsWaiting". This misdirects operators when only processInstancesActive setup fails. Change the message to identify processInstancesActive. (raw.githubusercontent.com)

Proposed fix

- zpn.logger.Error("Failed to register meter for jobsWaiting", "err", err)
+ zpn.logger.Error("Failed to register meter for processInstancesActive", "err", err)

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/cluster/partition/partition.go` around lines 87 - 102, Update the
error log for the processInstancesActive metric registration to identify
processInstancesActive instead of jobsWaiting. Locate the corresponding
registration and error-handling block in the partition metrics initialization,
leaving the metric setup and other log messages unchanged.